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Abstract Text (1) : 

A neural network is used in a system to detect abnormalities in cells, including 
cancer in bladder tissue cells. The system has an image analysis system for 
generating data representative of imaging variables from an image of stained cells. 
The set of data is provided to a neural network which has been trained to detect 
abnormalities from known tissue cells with respect to the data from the same set of 
imaging variables. A conventional sigmoid-activated neural network, or 
alternatively, a hybrid neural network having a combination of sigmoid, gaussian and 
sinusoidal activation functions may be utilized. The trained neural network applies 
a set of weight factors obtained during training to the data to classify the unknown 
tissue cell as normal or abnormal. 

Brief Summary Text (2) : 

The present invention relates, in general, to automated image recognition, and more 
particularly, to a neural network-based image recognition system for cancerous 
tissue cell detection. 

Brief Summary Text (6) : 

Significant progress in the detection and isolation of bladder tumor specific 
antigens has linked bladder cancer with an elevation of normal protein components in 
the plasma or urine of cancer patients. Thus, bladder cancer may be detected by 
identifying the abnormal presence of materials in the bladder cells. Since these 
tests are non-invasive, they could be routinely utilized to test those in high risk 
groups for early symptoms of bladder cancer. However, an approach using serum and 
plasma related components in urine appears to have limited usefulness in the early 
detection of bladder cancer as many of these identical components are also present 
in increased concentrations in urine from patients with non-neoplastic disease. 

Brief Summary Text (8) : 

Due to the vast amount of data to be processed, automation of the bladder cancer 
cell detection process is quite desirable. Various techniques have been proposed for 
the automated detection of cancer. Predominately, these prior attempts have relied 
on techniques such as feature extraction, template matching and other statistical or 
algorithmic methods. For instance, Melder and Koss described a decision tree 
representing the hierarchical classification scheme to classify extracted features 
from the triage of objects encountered in the urinary sediment. Karl K. Melder & 
Leopold G. Koss, "Automated Image Analysis in the Diagnosis of Bladder Cancer," 26 
Applied Optics 16, 3367 (1987) . Melder and Koss discussed the use of step-wise 
linear discriminant analysis in which features were automatically selected for the 
discriminant functions based on the pooled covariance matrix of more than sixty (60) 
cell features. Christen, et al . , discussed the application of a linear discriminant 
model from the SPSS/PC+ statistical package to the classification of cancerous 
cells. Christen, et al . , "Chromatin Texture Features in Hematoxylin and 
Eosin-Stained Prostate Tissue, " 16. Analytical and Quantitative Cytology and 
Histology, 16, 383 (1993) . 

Brief Summary Text (9) : 

Recently, artificial neural networks have been applied to the cancer detection 
process. This step is a logical extension of the pattern recognition capability of 
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artificial neural networks. Kunihiko Fukushima, "Neural Network Model for Selective 
Attention in Visual Pattern Recognition and Associative Recall, ■■ 26 Applied Optics 
23, 4985 (1987); Dwight D. Egbert, et al . , "Preprocessing of Biomedical Images for 
Neurocomputer Analysis," IEEE Int * 1 Conference on Neural Networks 1-561 (Jul. 24-27, 
1988) . 

Brief Summary Text (13) : 

Traditional back-propagation neural networks have been applied to bladder cells in 
Ciamac Moallemi, "Classifying Cells for Cancer Diagnosis Using Neural Network," 6 
IEEE Expert 6, 8 (1991) . Moallemi describes the application of a conventional neural 
network in the classification of noisy particles versus cell images, including 
cancerous and non-cancerous bladder cells. However, Moallemi does not teach the 
detection of malignant cells using a neural network. 

Brief Summary Text (14) : 

The application of neural networks to the classification of cytological specimens is 
discussed in U.S. Pat. No. 4,965,725 to Rutenberg. Rutenberg describes the use of a 
two-staged classifier system. The first classifier is a statistical classifier which 
identifies cell nuclei of interest by measurement of their integrated optical 
density, or nuclear stain density, defined as the sum of the pixel gray values for 
the object. Rutenberg discloses that, compared to normal cells, malignant cells tend 
to possess a larger, more densely staining nucleus. Based on the data provided by 
the primary classifier, Rutenberg further employs a neural network as a secondary 
classifier for evaluating the nucleus and its surrounding cytoplasm based on the 
observation that the ratio between the nucleus and the cytoplasm is an important 
indicator for malignant cell classification. However, Rutenberg does not utilize 
other predictive information such as the pgDNA value of a cell . 

Brief Summary Text (22) : 

In accordance with yet another aspect of the invention, a neural network is provided 
which detects cancerous cells by analyzing raw images of the cell and providing the 
imaging information derived from the pixels of the images to a neural network. 

Brief Summary Text (23) : 

In accordance with yet another aspect of the invention, a neural network is provided 
which* performs recognition of cancerous cells using information derived from an 
image of the cells, among others, the area, the average intensity, the shape, the 
texture, and the DNA content (pgDNA) of the cells. 

Brief Summary Text (24) : 

In accordance with yet another aspect of the invention, a neural network is provided 
which performs such recognition of cancerous cells using textural information 
derived from an image of the cells, among them angular second moment, contrast, 
coefficient of correlation, sum of squares, difference moment, inverse difference 
moment, sum average, sum variance, sum entropy, entry, difference variance, 
difference entropy, information measures, maximal correlation coefficient, 
coefficient of variation, peak transition probability, diagonal variance, diagonal 
moment, second diagonal moment, product moment, triangular symmetry 11 and blobness. 



Detailed Description Text (2) : 

FIG. 1 illustrates the image analysis 10 of the present invention. Initially, 
extracted unknown tissue cells are obtained from patients either from urine or a 
bladder wash. The urine or bladder wash at station 12 is poured through a 
urine/bladder wash filter 14 and cells are collected on the filter. The cells from 
wash filter 14 are placed on a slide 17 treated with poly-L-lysine and pressed 
gently to transfer cells to the slide as an imprint. The cells may also be placed on 
slides by means of a cytocentrif uge to ensure a reasonably flat distribution of 
cells with relatively few overlaps. 

Detailed Description Text (23) : 

After the cells have been chemically prepared, the first stage in the analysis of 
tissue cells for signs of cancer is the cell image acquisition in which the image of 
the sample tissue cell is digitized and separated from its background. The operation 
of the image analysis system 10 is described in more detail below. 
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Detailed Description Text (24) : 

After the cells have been chemically prepared, an image of the region of interest 16 
of the slide 17 containing the sample cells is captured and converted into data 
files using image acquisition system 18. Essentially, the slide 17 containing the 
sample cells is illuminated and a video camera in image acquisition 18 is focused on 
the chosen area in such a way that the frame occupies a maximal area of the screen 
of a monitor connected to the camera output. The output of the video camera is 
connected to a video digitizer 20, which converts the video encoding of an image 
into a raster (bit-mapped) format that can be processed by a computer. 

Detailed Description Text (25) : 

In one preferred embodiment, the CAS -200 RTM image analysis hardware and software 
(available from Cell Image Systems, Elmhurst, 111.) provides intensity information 
(at 8 bits per pixel) and a 256 . times . 256 pixel image , with a resolution of 
approximately 0.3 micron to 0.5 micron per pixel and a 229 gray, scale resolution. In 
another preferred embodiment, the Zeiss -IBAS RTM rapid scanning image analysis 
system (available from Roche Image Analysis Systems) provides a 1280 . times . 1024 
image , with a 255 gray scale capability. 

Detailed Description Text (26) : 

Because the conversion equipment alters the horizontal scale factor, in order to 
restore the aspect ratio distortion to 1:1, the original images are expanded 
horizontally by a factor of 1.78. After the aspect image ratio distortion 
correction, the cell images are tailored into an array of pixel images . The cell 
images are centrally aligned in this area. In the case of a large image, the image 
is clipped to fit into the given cell size within the boundary. 

Detailed Description Text (27) : 

In this manner, three sets of bladder cell images were obtained for classification 
by the neural network. Two sets of images were fluorescent-labeled with the M-344 
monoclonal antibody directed against a p3 00 bladder tumor associated antigen 
demonstrated to have clinical utility in identifying low grade bladder cancer cells 
in cytology specimens. One set of sixty (60) M-344 cell images were obtained at 
32. times, in a 60. times. 60 pixel array and included positive and false-positive 
images . The second set of eight hundred (800) M-344 cell images were collected at 
12. 5. times. magnification. Both sets of M-344 images were collected using an 
automated Zeiss -IBAS rapid scanning image analysis system. The third set of images 
covered the absorbance set of 467 cell images taken at 40 . times .magnifications of 
Feulgen stained cells. Images of the third set were captured using the CAS-200 image 
analysis system. 

Detailed Description Text (28) : 

In pattern recognition, only discriminate information contributes to correct 
classification, while information unrelated to the identification process does not 
and may even degrade performance. To reduce the input data and to focus the neural 
network on relevant variables, a pre-processing step is applied to extract cellular 
features from the raw image . Feature generation processing is used to extract 
information on the area, average intensity, shape (roundness) , texture and pgDNA 
values. In the preferred embodiment, an image analysis system 10 is used to extract 
the list of imaging variables from the digitized images . These imaging variables are 
stored in an input vector consisting of the following: 

Detailed Description Text (37) : 

Texture describes the interdependent characteristics of pixels within a neighboring 
area. Regular texture has more or less periodical patterns, while random texture is 
best described by its "coarseness." In the preferred embodiment, texture is measured 
by a convolution using a 3. times. 3 mask for extracting high frequency information 
(high pass filtering) from an image . The mask resembles a Laplacian kernel, commonly 
used for edge sharpening. Preferably, the filter G has a convolution mask of: 
##EQU4## 

Detailed Description Text (38) : 

This convolution mask is applied across every pixel in the interior area of a cell 
image , not including background. The texture for a particular cell can be obtained 
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as follows: ##EQU5## where 0(i,j) is the convolution output at location (i,j) on a 
cell image . G(i,j) corresponds to the intensity value at the location (i,j). .pi. 
represents the 8 neighboring locations of (i,j). 

Detailed Description Text (41) : 

In one embodiment, twenty-one nuclear descriptions based upon the gray- level 
transition probabilities of Feulgen stained nuclei are provided by the CAS System. 
The CAS constructs a "Markovian" matrix from a normalized cell nucleus image using 
an operator selected step size ranging from 1 pixel to 254 pixels . The step size 
defines the size of the "grain" (in number of pixels ) which is to be compared to the 
neighboring "grains". The default step size in the CAS System is 1 pixel, which was 
the step size used (i.e. every grain is equal to 1 pixel ) . In addition, each cell 
nucleus image is normalized by partitioning the image into eight equally frequent 
gray-levels, each level consisting of an equal number of pixels . This normalization 
process is done by first plotting the optical densities (gray- levels) above the 
operator set threshold against the number of pixels . This plot is divided into eight 
gray- level ranges ( optical density ranges) which contain an equal number of pixels . 
This results in a normalized cell nucleus image consisting of gray-level values from 
0-7 . 

Detailed Description Text (42) : 

An 8. times. 8 gray-level transition matrix is constructed from the normalized cell 
nucleus image by comparing the gray-levels of neighboring "grains" (i.e. if a given 
"grain" has a normalized value of 4, and its neighboring "grain" has a normalized 
value of 3, an entry in the matrix is made at location Row-r and Column-3) . This 
matrix is then transformed into an 8. times. 8 conditioned gray-level transition 
probability matrix by dividing every matrix entry by the total number of pixels in 
the cell nucleus image . This "Markovian" probability matrix is then used to compute 
the plurality of Markovian texture features. 

Detailed Description Text (46) : 

The pixel measurements within the cell object, if properly calibrated to represent 
optical density, relate the resulting values to such cell constituents such as the 
total amount of DNA in nucleated cells, the amount of monoclonal antibody attached 
to nuclear or cytoplasmic receptor cites, etc. In the instant invention, the 
measurements are calibrated to reflect the mass in picograms of these cell 
constituents. The DNA measurements are carried out on Feulgen stained preparations 
using optical density at a wavelength of 620 nm and with 256 gray scale levels. 

Detailed Description Text (47) : 

Each digitized image field may be described as a function n(x,y), where each value 
is a measured point of absorbance obeying the Beer-Lambert absorption law: ##EQU8## 

Detailed Description Text (48) : 

where I.sub.o is the incident light, I.sub.f is the transmitted light, .epsilon. is 
the specific extinction coefficient of the absorbing substance (.mu.m.sup.2 /pg) at 
a chosen wavelength (.lambda.) and with a specific bandpass (b) , m is the mass of 
the absorbing substance in picograms (pg) for each x,y image field measurement point 
inside the object or cell, and a is the measurement spot size in (.mu.m.sup.2). 
Assuming the boundary points have been properly determined for a single cell in the 
field of view, where the summation occurs only over the cell or cell part of 
interest, ##EQU9## 

Detailed Description Text (50) : 

Converting from transmitted light to optical density is necessary because the 
thickness of the object cells, or cell parts, can vary. This causes the light 
transmitted through any individual pixel to fall off exponentially as a function of 
the object thickness. The logarithm of the transmitted light converts this 
exponential function to a linear function and properly allows the summation of the 
pixels, whether they result from a part of the cell that is thinly spread or thickly 
condensed. 

Detailed Description Text (51) : 

After the features have been extracted from images cell by cell, computer 22 creates 
a feature vector data list. Items on the feature list are then normalized by their 
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corresponding standard deviations to prevent certain features that have 
substantially larger numerical values than others from dominating the training 
process. Even after normalization, certain cells still, have feature values out of 
the transition region, beyond which is a saturated region of activation function. To 
keep the feature value within the transition region, a scaling factor is used in the 
activation function of neurons on the texture processing. 

Detailed Description Text (52) : 

Once the feature vector is derived from the image acquisition system, the data is 
then processed by a neural network 30. A Sun, Sparcstation IPC workstation is used to 
train and test the neural network. The neural network 3 0 selected for the preferred 
embodiment of the present invention is a hybrid multilayer feed forward perceptron 
with error back-propagation training algorithm. The neural network 30 differs from 
conventional neural networks in that its architectural makeup consists of neuron 
elements with differing activation functions. 

Detailed Description Text (57) : 

The input layer 40 of the present invention comprises a plurality of nodes 42, 44, 
46 and 48, which are configured to receive input information from a computer 22 
outside of the neural network 30. The nodes of the input layer store data to be 
forward propagated to the neurons of the hidden layer 50. A hidden layer 50 
comprising a plurality of neurons 52, 62 and 72, accepts as input the output of the 
plurality of neurons 42, 44, 46 and 48 from the input layer 40. The neurons of the 
hidden layer 50 transmit its outputs to a plurality of neurons 92 and 100 in the 
output layer 90. The output layer 90 comprises a plurality of neurons 92 and 100 
that are configured to transmit an output pattern to an output source outside of the 
artificial neural network 30. One preferred embodiment employs 3,600 neurons in the 
input layer plus one neuron which holds a bias value to analyze the pixel 
information from the raw image of the cell. A second preferred embodiment employs 
five neurons in the input layer plus one bias neuron for analyzing an array of 
composite imaging variables of each cell. The second approach reduces the amount of 
information that must be processed by the neural network and thus improves the 
throughput of the neural network although at the expense of pre-processing 
computations. The neuron supplying the bias value is set to a constant value, such 
as one (1) . 

Detailed Description Text (60) : 

Referring now to FIG. 4, neuron 52 receives inputs from a plurality of nodes 42, 44, 
46 and 48. The interconnection between nodes 42, 44, 46 and 4 8 and neuron 52 have a 
plurality of weight factors 114, 116, 118 and 120 which adjust the value of the data 
coming from nodes 42, 44, 46 and 48. Weight factors 114, 116, 118 and 120 may be 
built from variables which can attenuate the incoming information appropriately. The 
inputs and the bias are supplied to multipliers which accept another input from a 
weight factor vector. Outputs from the multipliers are supplied to an adder unit for 
computing the weighted sum of the input values. In essence, the neurons perform a 
dot product of a vector of the weighted input values, including a bias value, with a 
vector representing the weights. The output value from the adder is passed through a 
non-linear function (sinusoid, sigmoid or gaussian) to generate a single neuron 
output value. The output of each neuron 52 in the input layer is propagated forward 
to each input of each neuron in the next layer, the hidden layer. In the case where 
the raw image data is fed directly to the neural network, the neurons of the input 
layer 40 scans 3,600 inputs which are neighboring input pixels, or gray values from 
an image . When the imaging variables are used, neurons of the input layer 40 scans 
its five input nodes. The feature inputs can be presented in any order. However, the 
order of presentation must match the order used for training. In using raw image 
data, the 3,600 gray scale levels are presented starting from upper left corner 
scanning to the right lower corner of the image . Using composite imaging data, the 
order of training data presentation must match the order used during the unknown 
sample data presentation. 

Detailed Description Text (85) : 

The network was first trained using data from the CAS-200 image analysis system. 
Table 1 reports the test results and overall classification rate on a 98 cell set. 
Values listed in Table 1 for each test run are: (1) the network output value; and 
(2) the square of the deviation [(error) .sup. 2 ] from the desired target output. An 
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output of indicates abnormality, while M 0" indicates normal cell structure. 
Detailed Description Text (86) : 

Table 1 results indicate that the network trained by raw cell images were capable of 
correctly classifying 72% of the test cells. However, using extracted features such 
as the features of area, intensity, texture and shape, the network was able to 
correctly classify 93% of the sample images . These data are most encouraging 
considering the fact that several low grade bladder carcinoma cells were included in 
the abnormal cell array. The positive results are most comparable if not better than 
those obtainable by an expert cytopathologist . 

Detailed Description Text (88) : 

The set of samples of M-344 cells was tested in three different configurations. In 
the first configuration, fifty percent (50%) of M-344 positive, negative and noise 
cell data were allocated for training, and the rest of data, including the entire 
false positive data, were used for testing. The individual classification results 
and overall rate were analyzed and the misclassif ied cells were determined. The 
threshold used for classification is 0.5. It was found that the feature vector 
trained network results in a higher average (taking all four groups into account) 
classification rate than the raw image trained one, 75.5% versus 69.8%. The score 
for false positive data is low, at about 55.7% and 57.6% for feature vector and raw 
image, respectively. But when calculated only for positive, negative and noise data, 
the classification rate increases to 88.4% and 77.8%. 

Detailed Description Text (90) : 

Neural networks were also studied in a third configuration when only positive and 
false-positive M-344 data are presented. Data were allocated evenly between training 
and testing. Next when only positive and false-positive were calculated, there was 
an increase in overall classification rate compared with configuration 1 for both 
feature vector and raw image trained networks. However, as far as the effect of 
false-positive cell identification participation of training is concerned, the 
results are consistent with that from configuration 2. 

Detailed Description Text (91) : 

Turning now to a more in-depth discussion of the present invention with specific 
re f erence to FIG. 1, the screening device is shown with particular emphasis on the 
classification elements embodied in the processing system. In the preferred 
embodiment, the processing system includes an image processor with digitizer and a 
neural network implemented on a general purpose processor. For improved recognition 
speed, the neural network can be implemented in hardware. 

Detailed Description Text (92) : 

The general processor is preferably a Sun SPARC IPC workstation although it may be 
another computer- type device suitable for efficient execution of the functions 
described herein such as an Intel Pentium or a Motorola PowerPC class computer 
system. The general processor controls the functioning of and the flow of data 
between components of the device, may cause execution of additional primary feature 
extraction algorithms, and handles the storage of image and classification 
information. The general processor additionally controls peripheral devices such as 
a printer, a storage device such as an optical or magnetic hard disk, a tape drive, 
etc., as well as other data collection devices. 

Detailed Description Text (94) : 

It will be appreciated by a person of ordinary skill in the art that, while 
particular resolutions and image sizes were described above, these particular values 
are exemplary. It will further be appreciated that different resolutions, image 
parameters, magnification levels, etc., can be employed to accomplish the same or 
similar results as the particular embodiment of the invention described above, and 
that all such differing resolutions, image parameters, imaging variable data items, 
neuron count, etc. are within the scope of the present invention. 

Detailed Description Text (95) : 

It should be further recognized that, although the preferred embodiment describes a 
neural network with one hidden layer, a plurality of hidden layers may be used. 
Also, different number of neurons may be used in input, output or hidden layers. In 
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addition, it should further be recognized that while the image processor with 
digitizer and the general processor are described operating in a serial manner, in 
actual practice as many functions will be performed in parallel as is possible. 
Hence, the components may process different slide segments or different areas of a 
segment concurrently, greatly reducing the time required to screen a sample. L/C"v" 
##SPC1## ##SPC2## ##SPC3## ##SPC4## ##SPC5## ##SPC6## ##SPC7## ##SPC8## ##SPC9## 
##SPC10## ##SPC11## ##SPC12## ##SPC13## ##SPC14## ##SPC15## ##SPC16## ##SPC17## 
##SPC18## ##SPC19## ##SPC20## ##SPC21## ##SPC22## ##SPC23## ##SPC24## ##SPC25## 
##SPC26## ##SPC27## ##SPC28## ##SPC29## ##SPC30## ##SPC31## ##SPC32## ##SPC33## 
##SPC34## ##SPC35## 
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